Diversity index

$$H\text{'}\; =\; -\backslash sum\_\{i=1\}^R\; p\_i\; \backslash ln(p\_i)$$

where is the proportion of characters belonging to the th type of letter in the string of interest. In ecology, is often the proportion of individuals belonging to the th species in the dataset of interest. Then the Shannon entropy quantifies the uncertainty in predicting the species identity of an individual that is taken at random from the dataset.

Although the equation is here written with natural logarithms, the base of the logarithm used when calculating the Shannon entropy can be chosen freely. Shannon himself discussed logarithm bases 2, 10 and , and these have since become the most popular bases in applications that use the Shannon entropy. Each log base corresponds to a different measurement unit, which has been called binary digits (bits), decimal digits (decits), and natural digits (nats) for the bases 2, 10 and , respectively. Comparing Shannon entropy values that were originally calculated with different log bases requires converting them to the same log base: change from the base to base is obtained with multiplication by .

The Shannon index () is related to the weighted geometric mean of the proportional abundances of the types. Specifically, it equals the logarithm of true diversity as calculated with :

$$H\text{'}\; =\; -\backslash sum\_\{i=1\}^R\; p\_i\; \backslash ln(p\_i)\; =\; -\backslash sum\_\{i=1\}^R\; \backslash ln\backslash left(p\_i^\{p\_i\}\backslash right)$$

This can also be written

$$H\text{'}\; =\; -\backslash left\backslash ln\backslash left(p\_1^\{p\_1\}\backslash right)$$

which equals

$$H\text{'}\; =\; -\backslash ln\backslash left(p\_1^\{p\_1\}p\_2^\{p\_2\}p\_3^\{p\_3\}\; \backslash cdots\; p\_R^\{p\_R\}\backslash right)\; =\; \backslash ln\; \backslash left\; (\; \{1\; \backslash over\; p\_1^\{p\_1\}p\_2^\{p\_2\}p\_3^\{p\_3\}\; \backslash cdots\; p\_R^\{p\_R\}\}\; \backslash right\; )\; =\; \backslash ln\; \backslash left\; (\; \{1\; \backslash over\; \{\backslash prod\_\{i=1\}^R\; p\_i^\{p\_i\}\}\}\; \backslash right\; )$$

Since the sum of the values equals 1 by definition, the denominator equals the weighted geometric mean of the values, with the values themselves being used as the weights (exponents in the equation). The term within the parentheses hence equals true diversity , and equals .

When all types in the dataset of interest are equally common, all values equal , and the Shannon index hence takes the value . The more unequal the abundances of the types, the larger the weighted geometric mean of the values, and the smaller the corresponding Shannon entropy. If practically all abundance is concentrated to one type, and the other types are very rare (even if there are many of them), Shannon entropy approaches zero. When there is only one type in the dataset, Shannon entropy exactly equals zero (there is no uncertainty in predicting the type of the next randomly chosen entity).

In machine learning the Shannon index is also called as Information gain.

$$\{\}^qH\; =\; \backslash frac\{1\}\{1-q\}\; \backslash ;\; \backslash ln\backslash left\; (\; \backslash sum\_\{i=1\}^R\; p\_i^q\; \backslash right\; )$$

which equals

$$\{\}^qH\; =\; \backslash ln\backslash left\; (\; \{1\; \backslash over\; \backslash sqrtq-1\}\}\; \backslash right\; )\; =\; \backslash ln(\{\}^q\backslash !D)$$

This means that taking the logarithm of true diversity based on any value of gives the Rényi entropy corresponding to the same value of .

The measure equals the probability that two entities taken at random from the dataset of interest represent the same type. It equals:

$$\backslash lambda\; =\; \backslash sum\_\{i=1\}^R\; p\_i^2,$$

where is richness (the total number of types in the dataset). This equation is also equal to the weighted arithmetic mean of the proportional abundances of the types of interest, with the proportional abundances themselves being used as the weights. Proportional abundances are by definition constrained to values between zero and one, but it is a weighted arithmetic mean, hence , which is reached when all types are equally abundant.

By comparing the equation used to calculate λ with the equations used to calculate true diversity, it can be seen that equals , i.e., true diversity as calculated with . The original Simpson's index hence equals the corresponding basic sum.

The interpretation of λ as the probability that two entities taken at random from the dataset of interest represent the same type assumes that the entities are sampled with replacement. If the dataset is very large, sampling without replacement gives approximately the same result, but in small datasets, the difference can be substantial. If the dataset is small, and sampling without replacement is assumed, the probability of obtaining the same type with both random draws is:

$$\backslash ell\; =\; \backslash frac\{\backslash sum\_\{i=1\}^R\; n\_i\; (n\_i\; -1)\}\{N\; (N-1)\}$$

where is the number of entities belonging to the th type and is the total number of entities in the dataset. This form of the Simpson index is also known as the Hunter–Gaston index in microbiology.

Since the mean proportional abundance of the types increases with decreasing number of types and increasing abundance of the most abundant type, λ obtains small values in datasets of high diversity and large values in datasets of low diversity. This is counterintuitive behavior for a diversity index, so often, such transformations of λ that increase with increasing diversity have been used instead. The most popular of such indices have been the inverse Simpson index (1/λ) and the Corrado Gini–Simpson index (1 − λ). Both of these have also been called the Simpson index in the ecological literature, so care is needed to avoid accidentally comparing the different indices as if they were the same.

$$\backslash frac\; 1\; \backslash lambda\; =\; \{1\; \backslash over\backslash sum\_\{i=1\}^R\; p\_i^2\}\; =\; \{\}^2D$$

This simply equals true diversity of order 2, i.e. the effective number of types that is obtained when the weighted arithmetic mean is used to quantify average proportional abundance of types in the dataset of interest.

The index is also used as a measure of the effective number of parties.